Knowledge-aware Leap-LSTM: Integrating Prior Knowledge into Leap-LSTM towards Faster Long Text Classification

While widely used in industry, recurrent neural networks (RNNs) are known to have deficiencies dealing with long sequences (e.g. slow inference, vanishing gradients etc.). Recent research has attempted accelerate RNN models by developing mechanisms skip irrelevant words input. Due the lack of labelled data, it remains as a challenge decide which skip, especially for low-resource classification tasks. In this paper, we propose Knowledge-AwareLeap-LSTM (KALL), novel architecture integrates prior human knowledge (created either manually or automatically) like in-domain keywords, terminologies lexicons into Leap-LSTM partially supervise skipping process. More specifically, knowledge-oriented cost function KALL; furthermore, two strategies integrate knowledge: (1) Factored KALL approach involves keyword indicator soft constraint skip-ping process, and (2) Gated enforces inclusion keywords while maintaining differentiable network training. Experiments on different public datasets show that our approaches are1.1x~2.6x faster than LSTM better accuracy 23.6x XLNet resource-limited CPU-only environment.